Skip to content

[WIP] T5v1.1 & MT5#8488

Closed
patrickvonplaten wants to merge 3 commits intohuggingface:masterfrom
patrickvonplaten:t5_v1_1
Closed

[WIP] T5v1.1 & MT5#8488
patrickvonplaten wants to merge 3 commits intohuggingface:masterfrom
patrickvonplaten:t5_v1_1

Conversation

@patrickvonplaten
Copy link
Contributor

@patrickvonplaten patrickvonplaten commented Nov 12, 2020

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to the it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten patrickvonplaten changed the title [T5v1.1 & MT5] add new t5 model [WIP] T5v1.1 & MT5 Nov 12, 2020
@patrickvonplaten patrickvonplaten mentioned this pull request Nov 12, 2020
Closed
3 tasks
@shenfe
Copy link
Contributor

shenfe commented Nov 12, 2020

Maybe wrong model config for T5.1.1. For instance, T5.1.1.small should have num_layers=8 and num_heads=6.

See https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/models/gin/models/t5.1.1.small.gin

shenfe
shenfe suggested changes Nov 12, 2020
View reviewed changes
Copy link
Contributor

@shenfe shenfe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, there are some problems I found:

  • different model configs, compared with the official
  • relu maybe unnecessary in FFN
  • lm_head weight should not share with the embedding of enc/dec

And just point it out if anything wrong :)

@patrickvonplaten
Copy link
Contributor Author

patrickvonplaten commented Nov 12, 2020

Maybe wrong model config for T5.1.1. For instance, T5.1.1.small should have num_layers=8 and num_heads=6.

See https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/models/gin/models/t5.1.1.small.gin

Thanks yeah, I implemented that.

The new model structure is now equal to mesh t5 v1.1.

If you download the t5v1.1 t5-small checkpoint and replace the corresponding path in check_t5_against_hf.py you can see that the models are equal.

There is still quite some work to do: write more tests, lots of cleaning and better design, and check if mT5 works with it.

@patrickvonplaten patrickvonplaten mentioned this pull request Nov 12, 2020
Closed
4 tasks
@shenfe
Copy link
Contributor

shenfe commented Nov 13, 2020

If you download the t5v1.1 t5-small checkpoint and replace the corresponding path in check_t5_against_hf.py you can see that the models are equal.

Hi, check_t5_against_hf.py still fails if I use a longer input text instead of Hello there, like Hello there. Let's put more words in more languages than I originally thought.

shenfe
shenfe reviewed Nov 13, 2020
View reviewed changes
check_t5_against_hf.py
loss = model(input_ids, labels=labels).loss
mesh_tf_loss = -(labels.shape[-1] * loss.item())

if mesh_tf_loss - score[0][0] < 1e-4:
Copy link
Contributor

@shenfe shenfe Nov 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better to use abs() here

Copy link
Contributor Author

@patrickvonplaten patrickvonplaten Nov 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will delete this file eventually - it's just for now :-)

@patrickvonplaten
Copy link
Contributor Author

patrickvonplaten commented Nov 13, 2020

If you download the t5v1.1 t5-small checkpoint and replace the corresponding path in check_t5_against_hf.py you can see that the models are equal.

Hi, check_t5_against_hf.py still fails if I use a longer input text instead of Hello there, like Hello there. Let's put more words in more languages than I originally thought.

Hmm, it works for me - do you experience that for T5v1.1 or mT5?

patrickvonplaten
patrickvonplaten commented Nov 13, 2020
View reviewed changes
src/transformers/modeling_t5v2.py
return self.weight * x


class T5v2DenseReluDense(nn.Module):
Copy link
Contributor Author

@patrickvonplaten patrickvonplaten Nov 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class differs from previous T5DenseReluDense significantly -> different weights are used here

patrickvonplaten
patrickvonplaten commented Nov 13, 2020
View reviewed changes
src/transformers/modeling_t5v2.py
)

sequence_output = decoder_outputs[0]
# Rescale output before projecting on vocab
Copy link
Contributor Author

@patrickvonplaten patrickvonplaten Nov 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For T5v1.1 there is no rescaling because the input and output embedding are not tied.

Copy link
Contributor

@shenfe shenfe Nov 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree on these changes. But the PyTorch T5.1.1 model still differs from the official tf version. I'm working on it too.

@shenfe
Copy link
Contributor

shenfe commented Nov 14, 2020

If you download the t5v1.1 t5-small checkpoint and replace the corresponding path in check_t5_against_hf.py you can see that the models are equal.

Hi, check_t5_against_hf.py still fails if I use a longer input text instead of Hello there, like Hello there. Let's put more words in more languages than I originally thought.

Hmm, it works for me - do you experience that for T5v1.1 or mT5?

Aha, the checking is OK now. Yesterday I made a mistake that when I changed the test input sentence in the check script, I didn't update the input length for MTF model from 4 to a longer value like 128. So actually the MTF model and PyTorch model received different inputs, and of course got different results.

Besides, if I add the z-loss to the CE loss at last, it differs from MTF score again. I just found MTF ignores z-loss when not training (code). So I think MTF model score does not include z-loss, but its training does, which is absent from HF T5 training. Well, this is absolutely not a blocking issue now.

Appreciate your great work :)

@patrickvonplaten
Copy link
Contributor Author

patrickvonplaten commented Nov 15, 2020

closing in favor of #8552.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@shenfe shenfe shenfe requested changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@patrickvonplaten @shenfe